System and method of establishing a multi-camera image using pixel remapping

ABSTRACT

A camera or vision system for establishing a composite image for displaying in a vehicle includes a first camera and a second camera and a controller. Each camera has a respective field of view that overlaps partially with the respective field of view of the other camera. Each camera has a respective imager for generating a respective preliminary digital image. The cameras together have a combined field of view. The controller is programmed to generate a final composite digital image that corresponds to a selected digital representation of the combined field of view of the cameras by using a remapping table to remap selected pixels from each of the preliminary digital images into selected positions of the final composite digital image. A plurality of methods for establishing a composite image are also provided.

FIELD OF THE INVENTION

The present invention relates to multi-camera systems for use invehicles, and more particularly multi-camera systems for use in vehicleswherein image manipulation is carried out on the images prior todisplaying the images to a vehicle occupant.

BACKGROUND OF THE INVENTION

There are few multi-camera systems currently available in vehicles. Suchsystems incorporate four cameras typically, and provide a vehicleoccupant with a composite image that is generated from the images takenby the four cameras. However, such systems can require a relativelylarge amount of processing power to generate the image that is displayedto the vehicle occupant, particular in situations where there ismanipulation of the images being carried out. Such manipulation of theimages may include dewarping, among other things.

It would be beneficial to provide a multi-camera system for a vehiclethat requires relatively little processing power.

SUMMARY OF THE INVENTION

In a first aspect, the invention is directed to a method of establishinga composite image for displaying in a vehicle, comprising:

a) providing a first camera and a second camera, each camera having afield of view;

b) positioning the cameras so that the fields of view of the camerasoverlap partially, wherein the cameras together have a combined field ofview;

c) recording preliminary digital images from the cameras, eachpreliminary digital image being made up of a plurality of pixels; and

d) generating a final composite digital image that corresponds to aselected digital representation of the combined field of view of thecameras by remapping selected pixels from each of the preliminarydigital images into selected positions of the final composite digitalimage.

In a second aspect, the invention is directed to a method ofestablishing a composite image for displaying in a vehicle, comprising:

a) providing a first camera and a second camera, a third camera and afourth camera, each camera having a field of view, wherein the camerastogether have a combined field of view that is a 360 degree field ofview around the vehicle;

b) positioning the cameras so that the field of view of each cameraoverlaps partially with the field of view of two of the other cameras;

c) recording preliminary digital images from the cameras, eachpreliminary digital image being made up of a plurality of pixels; and

d) generating a final composite digital image that corresponds to aselected digital representation of the combined field of view of thecameras by remapping selected pixels from each of the preliminarydigital images into selected positions of the final composite digitalimage,

wherein the preliminary digital images each have associated therewith apreliminary apparent camera viewpoint and the final composite digitalimage has associated therewith a final apparent camera viewpoint,

and wherein the selected pixels from the preliminary digital images areselected so that the final apparent camera viewpoint associated with thefinal composite digital image is higher than the preliminary apparentcamera viewpoints associated with the preliminary digital images,

and wherein the selected pixels from the preliminary digital images areselected so that any misalignment between the overlapping portions ofthe preliminary digital images is substantially eliminated,

and wherein the selected pixels from the preliminary digital images areselected so that the final composite digital image is dewarped ascompared to each of the preliminary digital images.

In a third aspect, the invention is directed to a system forestablishing a composite image for displaying in a vehicle, comprising afirst camera and a second camera and a controller. Each camera has afield of view that overlaps partially with the field of view of theother camera. Each camera has an imager for generating a preliminarydigital image. The cameras together have a combined field of view. Thecontroller is programmed to generate a final composite digital imagethat corresponds to a selected digital representation of the combinedfield of view of the cameras by using a remapping table to remapselected pixels from each of the preliminary digital images intoselected positions of the final composite digital image.

In a fourth aspect, the invention is directed to a method of generatinga remapping table for use in mapping pixels from a plurality ofpreliminary digital images into a final composite image, comprising:

a) driving a vehicle having a first camera and a second camera thereon,each camera having a field of view that overlaps partially with thefield of view of the other camera, each camera having an imager forgenerating one of the preliminary digital image, wherein the camerastogether have a combined field of view, wherein the vehicle furtherincludes a controller;

b) detecting a target feature along the path of the vehicle duringdriving, using the controller;

c) providing a first preliminary digital image from the first camera,wherein the first preliminary digital image contains a firstrepresentation of the target feature at a first point time;

d) determining the position of the first representation of the targetfeature in the first preliminary digital image;

e) providing a second preliminary digital image from the second camera,wherein the second preliminary digital image contains a secondrepresentation of the target feature at a second point time;

f) determining the position of the second representation of the targetfeature in the second preliminary digital image;

g) comparing the positions of the first and second representations ofthe target feature; and

h) generating at least one value for the remapping table based on theresult of the comparison in step g).

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will now be described by way of example only withreference to the attached drawings, in which:

FIG. 1 is a plan view of a vehicle with a camera system in accordancewith an embodiment of the present invention;

FIG. 2 is a schematic illustration of the camera system shown in FIG. 1;

FIGS. 3 a-3 d are images taken by cameras that are part of the camerasystem shown in FIG. 1;

FIG. 4 is a magnified view of the image shown in FIG. 3 d;

FIG. 5 is a composite final image generated by the camera system shownin FIG. 1;

FIG. 6 a is remapping table used to generate the final composite imageshown in FIG. 5 from the images shown in FIGS. 3 a-3 d;

FIG. 6 b is a graphical representation of the remapping that takes placeusing the remapping table shown in FIG. 6 a;

FIG. 6 c is a graphical representation of a step that takes place priorto the remapping illustrated in FIG. 6 b;

FIG. 7 is a plan view of a vehicle in a test area use to calibrate thecamera system shown in FIG. 1;

FIG. 8 a is a preliminary image from a camera from the camera systemshown in FIG. 1;

FIGS. 8 b and 8 c are images formed by progressive remapping of theimage shown in FIG. 8 a;

FIG. 8 d illustrates the analysis performed by the camera system shownin FIG. 1, to stitch together several remapped images;

FIG. 8 e is a final composite image generated using the analysis shownin FIG. 8 d;

FIGS. 9 a-9 c are remapping tables used to generate the images shown inFIGS. 8 b, 8 c and 8 e from the preliminary image shown in FIG. 8 a;

FIG. 10 shows target features on a road that can be used to assist incalibrating the camera system shown in FIG. 1 during driving;

FIG. 11 is a composite image formed using default remapping values,prior to the calibration of the camera system shown in FIG. 1 duringdrive; and

FIGS. 12 a-12 c are illustrations of events that would triggeradjustment of the remapping values used to generate the composite imageshown in FIG. 3.

DETAILED DESCRIPTION OF THE INVENTION

Reference is made to FIG. 1, which shows a vehicle 10 that includes avehicle body 12, and a multi-camera system 14 in accordance with anembodiment of the present invention. The multi-camera system 14 includesfour cameras 16 and a controller 18. The multi-camera system 14 isconfigured to display a composite image that is generated using all fourcameras 16 on an in-cabin display, shown at 20 in FIG. 2. The fourcameras 16 include a front camera 16F, a rear camera 16R, and driver'sside and passenger side cameras 16D and 16P.

Referring to FIG. 1, each camera 16 has a field of view 22. The field ofview of each camera 16 overlaps with the fields of view 22 of the twocameras 16 on either side of it. Preferably, the field of view of eachcamera 16 is at least about 185 degrees horizontally. Referring to FIG.2, each camera 16 includes an image sensor 24, which is used to generatea digital image taken from the camera's field of view 22. The imagesensor 24 may be any suitable type of image sensor, such as, for examplea CCD or a CMOS image sensor.

The digital image generated from the image sensor 24 may be referred toas a preliminary digital image, an example of which is shown at 26 inFIGS. 3 a-3 d. FIGS. 3 a-3 d show the preliminary digital images 26 fromthe four cameras 16. The images 26 are correspondingly identifiedindividually at 26F, 26R, 26D and 26P.

Each preliminary digital image 26 is made up of a plurality of pixels,which are shown at 28 in the magnified image shown in FIG. 4. It will benoted that the pixels 28 are enlarged in FIG. 4 for the sake of clarity.The actual image sensor 24 may have any suitable resolution. For exampleit may generate a digital image that is 640 pixels wide by 480 pixelshigh, or optionally an image that is 720 pixels wide×480 pixels high, oran image that is 1280 pixels wide×960 pixels high or even higher. Theoutput signals from the cameras 16 to the controller 18 may be in analogform such as in NTSC or PAL format, or in digital form using, forexample LVDS format, or Ethernet.

The controller 18 is programmed to generate a final composite digitalimage, shown at 30 in FIG. 5, that corresponds to a selected digitalrepresentation of the combined field of view of the cameras 16 by usinga remapping table 32 shown in FIG. 6 a to remap selected pixels 28 fromeach of the preliminary digital images 26 into selected positions of thefinal composite digital image 30.

The digital representation may incorporate one or more operations on theoriginal preliminary digital images 26. For example, the pixels 28 fromthe original images 26 may be remapped in such a way as to dewarp theimages 26. As can be seen in FIG. 5, the warpage present in the images26 is reduced (in this case it is substantially eliminated) in the finalcomposite digital image 30.

Another operation that may be carried out through the remapping isviewpoint adjustment. Each preliminary digital image 26 has associatedtherewith, an apparent viewpoint, which is the viewpoint from which thecamera 16 appears to have captured the image 26. In the preliminarydigital images 26, the apparent viewpoint of the camera 16 is the sameas the actual viewpoint of the camera 16 because no manipulation of theimage 26 has been carried out. However, it may be preferable, whenpresenting a 360 degree view around the vehicle to the vehicle driver,to present a bird's eye view. To accomplish this, the perspective of theimage is adjusted by adjusting the relative sizes of portions of thepreliminary images when remapping them to the final composite image 30.For example, the objects that are closer to the camera 16 appear largerin the image 26 than objects that are farther from the camera 16. Afterthe apparent viewpoint has been raised however, as shown in the finaldigital image 30, objects closer to the camera 16 are shrunk so thatthey are not larger than objects farther from the camera 16.

A graphical representation of the remapping that is carried out is shownin FIG. 6 b. In the exemplary embodiment of the present invention, thepreliminary digital images 26 were of sufficiently high resolution ascompared to the resolution of the final composite image 30 that there isnot a need for the controller to ‘stretch’ portions of the preliminaryimages 26 when generating the map for pixels in the final image 30. Inother words, in this particular embodiment, the controller 18 is notrequired to process a row of 10 pixels from the preliminary image 26 andconvert it to a row of 20 pixels in the final image 30. Thus, no pixelsin the final image 30 are ‘fabricated’ or generated by the controller18. Put another way, the preliminary images 26 are of sufficiently highresolution that the image manipulation that is carried out to arrive atthe final composite image 30 involves varying amounts of compression ofportions of the preliminary image (i.e. removing or skipping pixels),but does not involve stretching of any portions of the preliminary image(which could involve interpolating between pixels and thus ‘creating’pixels). It is conceivable, however, that the preliminary images wouldbe of relatively lower resolution such that the controller 18 would berelied upon in some instances to stretch portions of the preliminaryimages 26 when creating the final composite image 30. It will be notedthat in the exemplary embodiment, the resolution of each of thepreliminary images is 720 pixels wide by 480 pixels high, while theresolution of the final composite image is about 320 pixels wide by 480pixels high. As can be seen in the image in FIGS. 5 and 6 b, arepresentation of the vehicle 10 itself is inserted in the finalcomposite image 30. While it is preferred that none of the pixels in thefinal image 30 be ‘created’ through interpolation between adjacentpixels, it is contemplated that in certain situations some pixels may begenerated that way (i.e. by interpolating between adjacent pixels) so asto provide a relatively smooth transition between them.

Referring to FIG. 5, in the exemplary embodiment, given that the finalcomposite image 30 is only 320 pixels wide, a somewhat-dewarped rearview is also displayed on the display 20 for the vehicle driver,adjacent the 360 degree view.

Referring to FIG. 6 c, in some cases, the portion of the preliminarydigital image 26 from each individual camera that is used as part of thefinal composite image 30 may be a selected subset of the pixels of thepreliminary digital image 26. The particular subset used from eachpreliminary digital image is shown in a dashed box shown at 29 and willvary in position from camera to camera. It will be noted that the dashedbox represents the subset of pixels of the associated preliminarydigital image 26 that is involved in the generation of image 30, which,for greater certainty, is not to say that each pixel from subset 29necessarily will be a pixel in the image 30—rather it is to say that theimage 30 contains pixels that relate to or are taken from portion 29 andnot to the portion of the preliminary digital image that is outsideportion 29. The rest of the image pixels (i.e. the pixels that areoutside the portion 29 that is used to generate the composite image 30)are not needed and can be discarded. Only the pixels in the portions 29are streamed into the memory of the image engine (which is what themodule involved in generating the composite image 30 using the methodsdescribed herein may be referred to). By discarding those pixels thatare outside the portions 29, the memory bandwidth in image engine can bereduced, so that a slower memory can be utilized which may advantages interms of reducing system cost, and/or increasing reliability.

Aspects of the calibration of the multi-camera system 14 will now bediscussed. This calibration is used in order to assist in determiningthe remapping values in the remapping table 32 (FIG. 6 a).

Initially, the cameras 16 are mounted to the vehicle body 12 and thevehicle 10 is positioned at a location (as shown in FIG. 7) whereatthere is a predetermined test arrangement 34 of alignment landmarks 36,and dewarping landmarks 38.

In the exemplary test arrangement 34 shown in FIG. 7, it can be seenthat the landmarks 38 are straight lines. The preliminary digital imagefrom one of the cameras 16 (e.g., the rear camera) is shown at 40 inFIG. 8 a. Three functions are carried out on the preliminary digitalimages 40 to prepare the final composite image 30 shown in FIG. 8 e. Thefunctions are: dewarping, viewpoint adjustment, and offset correction.These functions may be carried out sequentially, and an intermediateremapping table may be generated in association with each function.Referring to FIG. 8 a, it can be seen that there is substantial warpingin the representations 42 of the landmarks 38 in the preliminary digitalimage 40. Knowing that the actual landmarks 38 are straight lines, thiswarping can be compensated for when determining the remapping of thepixels from the preliminary digital image 40 into the dewarpedintermediate image shown at 44 in FIG. 8 b. As can be seen, therepresentations shown at 45 of the landmarks 38 are dewarpedsubstantially completely. The remapping necessary to generate thedewarped image 44 may be stored in a first intermediate remapping tableshown at 46 in FIG. 9 a. It will be understood that a preliminarydigital image 40 from each camera 16 will be dewarped to generate adewarped image 44 and so four first intermediate remapping tables 46will be generated (i.e. one table 46 for each camera 16).

The dewarped image 44 may then be viewpoint adjusted so as to move theapparent viewpoint of the camera 16 upwards to generate a resulting‘dewarped and viewpoint-adjusted’ image 48 in FIG. 8 c, using a secondintermediate remapping table shown at 49 in FIG. 9 b. The remapping datato be inserted in the second intermediate remapping table 49 may begenerated relatively easily by determining what adjustments need to beapplied to the longitudinal representations 45 a to make them parallelto each other, what adjustments need to be applied to the transverserepresentations 45 b to make them parallel to each other (in this casevirtually no adjustment in that regard is required), what adjustmentsneed to be applied to the representations 45 so that they are spacedappropriately from each other, and to make the longitudinalrepresentations 45 a extend perpendicularly to the transverserepresentations 45 b, so as to match the known angles at which theactual longitudinal landmarks 38 intersect with the actual transverselandmarks 38. It will be understood that each image 44 will beviewpoint-adjusted to generate a dewarped and viewpoint-adjusted image48 and so four second intermediate remapping tables 49 will begenerated. The representations of the landmarks 38 in FIG. 8 c are shownat 50.

The dewarped and viewpoint-adjusted image 48 shown in FIG. 8 c from onecamera 16 may be compared to the other dewarped and viewpoint-adjustedimages 48 from the other cameras 16 to determine whether there is anyoffset adjustment necessary. This comparison is illustrated in FIG. 8 d.The versions of the images 48 shown in FIG. 8 d have been greatlysimplified and only include a few representations 50 of landmarks 38 andrepresentations 54 of landmarks 36, so as to facilitate explanation andillustration of the comparison that is being carried out. It will beunderstood however, that the actual comparison that is carried out maybe done with all of the representations 50 in the images 48 shown inFIG. 8 c.

As can be seen in FIG. 7, the alignment landmarks 36 are arranged ingroups 52, shown individually at 52 a, 52 b, 52 c and 52 d. Each group52 is visible to at least two of the cameras 16. As shown in FIG. 8 d,each image 48 contains representations 53 of some of the alignmentlandmarks 36. The groups of representations are identified at 54. It canbe seen that the images shown at 48F (front) and 48D (driver's side)both contain representations 54 of the group 52 a of landmarks 36.Similarly the images shown at 48F and 48P (passenger side) both containrepresentations 54 of the group 52 b of landmarks 36. Similarly theimages shown at 48P and 48R (rear) both contain representations 54 ofthe group 52 c of landmarks 36. Finally, the images shown at 48R and 48Dboth contain representations 54 of the group 52 d of landmarks 36. An Xaxis and a Y axis are shown at 56 and 58 respectively in FIG. 8 d. The Xaxis and Y axis offsets between the representations 54 of group 52 a inimage 48F and the representations 54 of group 52 a in image 48D aredetermined, and these offsets can be taken into account when remappingpixels from these two images 48 into the final composite image 30 shownin FIG. 8 e to ensure that the final composite image 30 transitionssmoothly from pixels taken from image 48F to pixels taken from image48D. Similarly, the offsets can easily be determined between therepresentations 54 shown in any two adjacent images 48, and thisinformation can be taken into account when remapping the pixels from theimages 48 into the final composite image 30. The remapping informationfrom the images 48 to the final composite image 30 may be stored in athird intermediate remapping table 60 shown in FIG. 9 c. It will beunderstood that only a single remapping table 60 is generated, whichremaps pixels from each of the four images 48 into the final compositeimage 30.

Once the four first remapping tables 46, the four second remappingtables 49 and the third remapping table 60 are generated, the remappingtable 32 shown in FIG. 6 a can be generated by combining the remappinginformation in all these tables 46, 49 and 60. Once generated, theremapping table 32 may be stored in the permanent storage memory (shownat 80 in FIG. 2) that is part of the camera system 14.

However, the controller 18 may additionally store one or more of theindividual remapping tables for use in generating and displaying anintermediate image. For example, it may be desired to show a dewarpedrear view from the vehicle 10 in some instances, such as when the driveris backing the vehicle 10 up. The preliminary digital image 40 from therear camera 16 can be remapped quickly and easily using the firstintermediate remapping table 46 to generate the dewarped rear view image44. Other viewing modes are also possible and would benefit from havingone or more of the intermediate remapping tables stored in the memory80. For example, a split view showing images from the driver's side andpassenger side cameras could be provided.

In the above example, the test arrangement 34 of landmarks 36 and 38were provided as images painted on the floor of an indoor test area. Itwill be noted that other means of providing the test arrangement 34 canbe provided. For example, the test arrangement can be provided on matsplace on the floor of the test area. Alternatively, the test arrangement34 could projected on the floor of the test area using any suitablemeans, such as one or more lasers, or one or more projectors, or somecombination of both.

In the example described above, four cameras are used to generate a 360degree view around the vehicle, using pixel remapping. It will beunderstood that the advantages of pixel remapping are not limited tocamera systems that employ four cameras. For example, in an alternativeembodiment that is not shown, the vehicle may include cameras 16 mountedat each of the front corners and each of the rear corners of thevehicle. Depending on whether the vehicle is leaving a parking spot bydriving forward or by backing up, the two front corner cameras or thetwo rear corner cameras could be used to form a view that showscross-traffic in front and to the sides of the vehicle, or behind and tothe sides of the vehicle depending on whether the vehicle is drivingforward or backing up. In such an embodiment, a final composite imagecan be generated using pixel remapping, but would be generated based onimages from only two cameras (i.e. the cameras at the two front cornersof the vehicle, or alternatively the cameras at the rear two corners ofthe vehicle).

It will be noted that, while the lines 38 in the test arrangement havebeen shown as straight lines, they need not be. They may be any suitableselected shape, which is then compared to its representation in theimages 40 and 44 to determine how to remap the pixels to reduce warpingand to carry out viewpoint adjustment.

In the test arrangement 34 shown in FIG. 7 the alignment landmarks 36are the intersections between the lines 38. It will be understoodhowever that the alignment landmarks could be other things, such as, forexample, a group of unconnected dots arranged in a selected arrangement(e.g., arranged to form a square array).

The above description relates to the calibration of the camera system 14in a controlled environment using a test arrangement 34 to generate theremapping table 32 for storage in the memory 80.

It may be desirable to permit the controller 18 to calibrate orrecalibrate the camera system 14 during driving. To do this, thecontroller 18 identifies a target feature that appears in an image fromone of the cameras 16. The target feature is shown in FIG. 10 at 61 andmay be, for example, a crack in the pavement, a lane marker or a pieceof gravel. In FIG. 10 numerous examples of possible target features areshown, although the controller 18 need only work with one target feature61 that will pass on one side of the vehicle, in order to calibratethree of the cameras 16 to each other (i.e. the front camera, the cameraon whichever side of the vehicle that the target feature 61 will pass,and the rear camera). At least one target feature 61 needs to beidentified that will pass on the other side of the vehicle 10 (althoughnot necessarily at the same time as the first target feature 61), inorder to calibrate the camera on the other side of the vehicle to theother three cameras.

As the vehicle 10 is driven (preferably below a selected speed) past thetarget feature 61, the target feature 61 moves through the field of viewof the front camera 16, through the field of view of one of the sidecameras 16 and finally through the field of view of the rear camera 16.A representation of the target feature 61 will thus move through imagesfrom the front camera 16, then through images from one of the sidecameras 16, and then through images from the rear camera 16. Byanalyzing the movement of the representation of the target feature 61(e.g. its position, its direction of travel and its speed of movement)particularly as it transitions from images from one camera into theimages from a subsequent camera the controller 18 can determine X and Yoffsets, angular offsets, differences in scale, and possibly otherdifferences, between images from one camera and another. This analysismay be carried out as follows: The controller 18 may start with adefault set of remapping values for the remapping table 32 to generate afinal composite image 30 from the four images. This default set ofremapping values may be based on a simple algorithm to crop thepreliminary digital images as necessary, rotate them as necessary andscale them as necessary to fit them in allotted zones 63 (shownindividually at 63F, 63R, 63D and 63P) of a preliminary composite image65 shown in FIG. 11. Optionally the default remapping values may alsoachieve dewarping and viewpoint adjustment, based on informationobtained during testing in a test area similar to the test area shown inFIG. 7. Alternatively, the default remapping values may be the values inthe remapping table 32 from a previous calibration (e.g. a calibrationperformed at a test area shown in FIG. 7, or a previous calibrationperformed during driving). As shown in FIG. 11, demarcation lines shownat 67 show the boundaries between the zones 63.

FIGS. 12 a, 12 b and 12 c show two adjacent zones 63 and the demarcationline 67 between them, to illustrate the analysis of the movement of therepresentation of the target feature 61. The adjacent zones in FIGS. 12a, 12 b and 12 c, are zones 63D and 63R. It will be understood however,that these figures are provided solely to illustrate the analysis thatis carried out by the controller 18 on the movement of therepresentation of the target feature 61 between all applicable pairs ofadjacent zones 63.

In FIGS. 12 a-12 c, the representation is shown at 69 and is shown attwo different instants in time in each of the FIGS. 12 a-12 c. Theposition of the representation 69 at the first, earlier instant of timeis shown at 70 a, and at the second, later instant of time at 70 b. Atthe first instant in time, the representation 69 is in the zone 63D. Atthe second instant of time, the representation 69 is in the zone 63R.

While tracking the movement of the representation 69, if the controller18 detects that the representation 69 shifts horizontally by some amountof pixels (as shown in FIG. 12 a) as it crosses the demarcation line 67(by comparing the positions 70 a and 70 b of the representation 69 atthe two instants in time), then the controller 18 can adjust theremapping values accordingly for one or both of the images that aremapped to the zones 63D and 63R.

With reference to FIG. 12 b, while tracking the movement of therepresentation 69, the controller 18 may store an expected position 70 cfor the representation 69 at the second instant of time, based on thespeed and direction of travel of the representation 69. The controller18 may compare the actual detected position 70 b of the representation69 at the second instant of time with the expected position 70 c of therepresentation 69 at the second instant of time, and, if there is avertical offset, the controller 18 can adjust the remapping valuesaccordingly for one or both of the images that are mapped to the zones63D and 63R.

With reference to FIG. 12 c, while tracking the movement of therepresentation 69, if the controller 18 detects that the representation69 changes its direction of travel by some angle as it crosses thedemarcation line 67 (by deriving a first direction of travel based onpositions 70 a and 70 a′, deriving a second direction of travel based onpositions 70 b and 70 b′, and by comparing the two directions oftravel), then the controller 18 can adjust the remapping valuesaccordingly for one or both of the images that are mapped to the zones63D and 63R.

It may be that only the remapping values associated with pixels in theimmediate vicinity of the representation 69 are adjusted. Thus, thevehicle 10 may drive along while the controller 18 scans for and detectstarget features 61 at different lateral positions on the road, so thatdifferent portions of the remapping table 32 are adjusted. As analternative way, the vehicle 10 may drive along while the controller 18scans for and detects multiple target features 61 at different lateralpositions across each demarcation line 67. At a selected point in time(e.g., after having detected target features 61 over a selected amountof lateral positions along the demarcation line 67), the controller 18may then determine a formula (or set of formulas) that could be used toremap the entire area along the demarcation line 67 as a whole, based onthe changes in the positions of the representations 69. Then thecontroller 18 uses that formula (or set of formulas) to remap the entirearea-around the border. For greater certainty the formula or formulasmay be linear or nonlinear.

After detecting a target feature 61 at a particular lateral position onthe road, and adjusting a portion of the remapping table 32 through thetechniques described above, the controller 18 may also scan for anddetect a second target feature 61 at approximately the same lateralposition on the road and apply these techniques again, in order toimprove the accuracy of the adjustments to the remapping values.

In many situations (e.g. after a malfunctioning or damaged camera hasbeen replaced in the vehicle or simply due to a shift in the position ofa camera over time in the vehicle) it may be that a camera is no longerin the same position and orientation as it was before. As a result,during the calibration procedure some pixels will require a change intheir remapping due to new changes that occur to representations 69 asthey cross demarcation lines 67. If the changes to the remapping areonly carried out in the immediate vicinity of the affected pixels thenthere will be a misalignment of those pixels with other pixels that arenot changed. If the changes are made to all the pixels in an image 26then this could cause a problem with the remapping of pixels at theother demarcation line 67 at the other end of the image 26. To addressthis issue, when a new remapping is carried out on a selected pixel, theremapping is carried out in progressively diminishing amounts on a rangeof adjacent pixels. For example, if during a calibration it isdetermined that a particular pixel should be shifted 5 pixels laterally,a selected first number of pixels longitudinally adjacent to that pixelwill be shifted 5 pixels laterally, a selected second number of pixelslongitudinally adjacent to the first number of pixels will be shifted 4pixels laterally, a selected third number of pixels adjacent to thesecond number of pixels will be shifted 3 pixels laterally, and so onuntil there is no lateral shift to carry out. This effectively smoothesout the remapping of the pixels, as an example, in a car wherein thefront camera is damaged in a traffic accident, and is replaced, arecalibration will be carried out, and the controller 18 may detect thatthe remapping that applied at the front left and right demarcation lines67 does not work anymore. The controller 18 may determine a newremapping for these regions. However, the remapping that occurs at therear left and right demarcation lines is still good, since the left,right and rear cameras have not been moved. To address this, thecontroller 18 may remap some selected number of pixels (e.g. 50 pixels),rearward of the newly remapped pixels along the front left and rightdemarcation lines 67 in groups by progressively smaller amountseventually reducing the remapping to zero. No remapping of pixels takesplace along the rear left and right demarcation lines 67.

After a selected period of time of driving, or after detecting enoughtarget features at enough lateral positions to ensure that a sufficientamount of adjustment of the remapping table has been made, thecontroller 18 may end the calibration process.

The particular cameras 16 that are used in the camera system 14 may beany suitable cameras. One example of an acceptable camera is a ReversAid camera made by Magna Electronics, an operating unit of MagnaInternational Inc. of Aurora, Ontario, Canada.

The camera or vision system includes a display screen that is incommunication with a video line and that is operable to display imagescaptured by the camera or camera module. The display screen may bedisposed in an interior rearview mirror assembly of the vehicle, and maycomprise a video mirror display screen, with video information displayedby the display screen being viewable through a transflective mirrorreflector of the mirror reflective element of the interior rearviewmirror assembly of the vehicle. For example, the camera or camera modulemay be disposed at a rearward portion of the vehicle and may have arearward facing field of view. The display screen may be operable todisplay images captured by the rearward viewing camera during areversing maneuver of the vehicle.

Surround view/panoramic vision/birds-eye vision multi-camera systems areknown, such as described in U.S. Pat. Nos. 6,275,754; 6,285,393;6,483,429; 6,498,620; 6,564,130; 6,621,421; 6,636,258; 6,819,231;6,917,378; 6,970,184; 6,989,736; 7,012,549; 7,058,207; 7,071,964;7,088,262; 7,145,519; 7,161,616; 7,230,640; 7,248,283; 7,280,124;7,295,227; 7,295,229; 7,301,466; 7,317,813; 7,369,940; 7,463,281;7,468,745; 7,519,459; 7,592,928; 7,680,570; 7,697,027; 7,697,029;7,742,070; 7,768,545; and/or 7,782,374, and/or U.S. Publication Nos.2003/0137586; 2005/0030379; 2005/0174429; 2005/0203704; 2007/0021881;2007/0165909; 2008/0036857; 2008/0144924; 2009/0179773; and/or2010/0013930, and/or International Publication Nos. WO2000/064175;WO2005/074287; WO2007/049266; WO2008/044589; WO2009/095901;WO2009/132617; and/or WO2011/014482, and/or European Pat. PublicationNos. EP1022903; EP1179958; EP 1197937; EP1355285; EP1377062; EP1731366;and/or EP1953698, and/or MURPHY, TOM, “Looking Back to the Future—Howhard can it be to eliminate a driver's blindspot?”, Ward's AutoWorld,May 1, 1998, which are all hereby incorporated herein by reference intheir entireties. Such systems benefit from the present invention.

The video display is operable to display a merged or composite image toprovide a panoramic or surround view for viewing by the driver of thevehicle. The vision system may utilize aspects of the vision and displaysystems described in U.S. Pat. Nos. 5,550,677; 5,670,935; 6,498,620;6,222,447; and/or 5,949,331, and/or PCT Application No.PCT/US2011/061124, filed Nov. 17, 2011, and/or PCT Application No.PCT/US2010/025545, filed Feb. 26, 2010 and published on Sep. 2, 2010 asInternational Publication No. WO 2010/099416, which are herebyincorporated herein by reference in their entireties.

Optionally, the video display may display other images, and may displaya surround view or bird's-eye view or panoramic-view images orrepresentations at the display screen, such as by utilizing aspects ofthe display systems described in PCT Application No. PCT/US10/25545,filed Feb. 26, 2010 and published Sep. 2, 2010 as InternationalPublication No. WO 2010/099416, and/or PCT Application No.PCT/US10/47256, filed Aug. 31, 2010 and published Mar. 10, 2011 asInternational Publication No. WO 2011/028686, and/or U.S. provisionalapplications, Ser. No. 61/540,256, filed Sep. 28, 2011; Ser. No.61/466,138, filed Mar. 22, 2011; Ser. No. 61/452,816, filed Mar. 15,2011; and Ser. No. 61/426,328, filed Dec. 22, 2010, which are all herebyincorporated herein by reference in their entireties. Examples of bird'seye view systems and associated techniques are described in U.S. Pat.Nos. 5,670,935; 6,636,258; 7,145,519; 7,161,616; 7,230,640; 7,248,283;7,295,229; 7,301,466; and/or 7,592,928, and/or International PublicationNo. WO 2010/099416, published Sep. 2, 2010, and/or PCT Application No.PCT/US10/47256, filed Aug. 31, 2010 and published Mar. 10, 2011 asInternational Publication No. WO 2011/028686, which are herebyincorporated herein by reference in their entireties. Optionally, thecamera and video display may operate to display other images, and maydisplay a trailer angle or the like of a trailer behind the vehicle.

The vision display system may operate to display the rearward images atthe video mirror display, and may do so responsive to the driver of thevehicle shifting the vehicle into a reverse gear (such as by utilizingaspects of the vision systems described in U.S. Pat. Nos. U.S. Pat. Nos.5,550,677; 5,670,935; 6,498,620; 6,222,447; and/or 5,949,331, and/or PCTApplication No. PCT/US2011/056295, filed Oct. 14, 2011, which are herebyincorporated herein by reference in their entireties).

Optionally, the system of the present invention may utilize aspects ofthe vision systems and lane departure systems and/or lane change aidsand/or side object detection systems of the types described in U.S. Pat.Nos. 7,914,187; 7,720,580; 7,526,103; 7,038,577; 7,004,606; 6,946,978;6,882,287; and/or 6,396,397, and/or PCT Application No.PCT/US2011/059089, filed Nov. 3, 2011, which are hereby incorporatedherein by reference in their entireties.

The imaging sensor or camera that captures the image data for imageprocessing may comprise any suitable camera or sensing device, such as,for example, an array of a plurality of photosensor elements arranged in640 columns and 480 rows (a 640×480 imaging array), with a respectivelens focusing images onto respective portions of the array. Thephotosensor array may comprise a plurality of photosensor elementsarranged in a photosensor array having rows and columns. The camera orimaging sensor and/or the logic and control circuit of the imagingsensor may function in any known manner, such as by utilizing aspects ofthe vision or imaging systems described in U.S. Pat. Nos. 6,806,452;6,690,268; 7,005,974; 7,123,168; 7,004,606; 6,946,978; 7,038,577;6,353,392; 6,320,176; 6,313,454; 6,824,281; 5,550,677; 5,877,897;6,498,620; 5,670,935; 5,796,094; and/or 6,396,397, and/or PCTApplication No. PCT/US2010/028621, filed Mar. 25, 2010, which are allhereby incorporated herein by reference in their entireties.

The imaging device and control and image processor and any associatedillumination source, if applicable, may comprise any suitablecomponents, and may utilize aspects of the cameras and vision systemsdescribed in U.S. Pat. Nos. 5,550,677; 5,877,897; 6,498,620; 5,670,935;5,796,094; 6,396,397; 6,806,452; 6,690,268; 7,005,974; 7,123,168;7,004,606; 6,946,978; 7,038,577; 6,353,392; 6,320,176; 6,313,454; and6,824,281, and/or International Publication No. WO 2010/099416,published Sep. 2, 2010, and/or PCT Application No. PCT/US10/47256, filedAug. 31, 2010, and/or U.S. patent application Ser. No. 12/508,840, filedJul. 24, 2009, and published Jan. 28, 2010 as U.S. Pat. Publication No.US 2010-0020170, which are all hereby incorporated herein by referencein their entireties. The camera or cameras may comprise any suitablecameras or imaging sensors or camera modules, and may utilize aspects ofthe cameras or sensors described in U.S. patent application Ser. No.12/091,359, filed Apr. 24, 2008 and published Oct. 1, 2009 as U.S.Publication No. US-2009-0244361, and/or U.S. Pat. Nos. 7,965,336 and/or7,480,149, which are hereby incorporated herein by reference in theirentireties. The imaging array sensor may comprise any suitable sensor,and may utilize various imaging sensors or imaging array sensors orcameras or the like, such as a CMOS imaging array sensor, a CCD sensoror other sensors or the like, such as the types described in U.S. Pat.Nos. 7,965,336; 5,550,677; 5,670,935; 5,760,962; 5,715,093; 5,877,897;6,922,292; 6,757,109; 6,717,610; 6,590,719; 6,201,642; 6,498,620;5,796,094; 6,097,023; 6,320,176; 6,559,435; 6,831,261; 6,806,452;6,396,397; 6,822,563; 6,946,978; 7,339,149; 7,038,577; 7,004,606; and/or7,720,580, and/or PCT Application No. PCT/US2008/076022, filed Sep. 11,2008 and published Mar. 19, 2009 as International Publication No.WO/2009/036176, and/or PCT Application No. PCT/US2008/078700, filed Oct.3, 2008 and published Apr. 9, 2009 as International Publication No.WO/2009/046268, which are all hereby incorporated herein by reference intheir entireties.

The camera module and circuit chip or board and imaging sensor may beimplemented and operated in connection with various vehicularvision-based systems, and/or may be operable utilizing the principles ofsuch other vehicular systems, such as a vehicle headlamp control system,such as the type disclosed in U.S. Pat. Nos. 5,796,094; 6,097,023;6,320,176; 6,559,435; 6,831,261; 7,004,606; 7,339,149; and/or 7,526,103,which are all hereby incorporated herein by reference in theirentireties, a rain sensor, such as the types disclosed in commonlyassigned U.S. Pat. Nos. 6,353,392; 6,313,454; 6,320,176; and/or7,480,149, which are hereby incorporated herein by reference in theirentireties, a vehicle vision system, such as a forwardly, sidewardly orrearwardly directed vehicle vision system utilizing principles disclosedin U.S. Pat. Nos. 5,550,677; 5,670,935; 5,760,962; 5,877,897; 5,949,331;6,222,447; 6,302,545; 6,396,397; 6,498,620; 6,523,964; 6,611,202;6,201,642; 6,690,268; 6,717,610; 6,757,109; 6,802,617; 6,806,452;6,822,563; 6,891,563; 6,946,978; and/or 7,859,565, which are all herebyincorporated herein by reference in their entireties, a trailer hitchingaid or tow check system, such as the type disclosed in U.S. Pat. No.7,005,974, which is hereby incorporated herein by reference in itsentirety, a reverse or sideward imaging system, such as for a lanechange assistance system or lane departure warning system or for a blindspot or object detection system, such as imaging or detection systems ofthe types disclosed in U.S. Pat. Nos. 7,881,496; 7,720,580; 7,038,577;5,929,786 and/or 5,786,772, which are hereby incorporated herein byreference in their entireties, a video device for internal cabinsurveillance and/or video telephone function, such as disclosed in U.S.Pat. Nos. 5,760,962; 5,877,897; 6,690,268; and/or 7,370,983, and/or U.S.patent application Ser. No. 10/538,724, filed Jun. 13, 2005 andpublished Mar. 9, 2006 as U.S. Publication No. US-2006-0050018, whichare hereby incorporated herein by reference in their entireties, atraffic sign recognition system, a system for determining a distance toa leading or trailing vehicle or object, such as a system utilizing theprinciples disclosed in U.S. Pat. Nos. 6,396,397 and/or 7,123,168, whichare hereby incorporated herein by reference in their entireties, and/orthe like.

Optionally, the circuit board or chip may include circuitry for theimaging array sensor and or other electronic accessories or features,such as by utilizing compass-on-a-chip or EC driver-on-a-chip technologyand aspects such as described in U.S. Pat. No. 7,255,451 and/or U.S.Pat. No. 7,480,149; and/or U.S. patent applications, Ser. No.11/226,628, filed Sep. 14, 2005 and published Mar. 23, 2006 as U.S.Publication No. US-2006-0061008, and/or Ser. No. 12/578,732, filed Oct.14, 2009 and published Apr. 22, 2010 as U.S. Publication No.US-2010-0097469, which are hereby incorporated herein by reference intheir entireties.

Optionally, the vision system may include a display for displayingimages captured by one or more of the imaging sensors for viewing by thedriver of the vehicle while the driver is normally operating thevehicle. Optionally, for example, the vision system may include a videodisplay device disposed at or in the interior rearview mirror assemblyof the vehicle, such as by utilizing aspects of the video mirror displaysystems described in U.S. Pat. No. 6,690,268; 7,370,983; 7,329,013;7,308,341; 7,289,037; 7,249,860; 7,004,593; 4,546,551; 5,699,044;4,953,305; 5,576,687; 5,632,092; 5,677,851; 5,708,410;5,737,226;5,802,727; 5,878,370; 6,087,953; 6,173,508; 6,222,460; 6,513,252;5,530,240; 6,329,925; 7,855,755; 7,626,749; 7,581,859; 7,446,650;7,446,924; 7,370,983; 7,338,177; 7,274,501; 7,255,451; 7,195,381;7,184,190; 5,668,663; 5,724,187; 7,338,177; 5,910,854; 6,420,036; and/or6,642,851, and/or European patent application, published Oct. 11, 2000under Publication No. EP 0 1043566, and/or PCT Application No.PCT/US2011/056295, filed Oct. 14, 2011, and/or U.S. patent applications,Ser. No. 11/226,628, filed Sep. 14, 2005 and published Mar. 23, 2006 asU.S. Publication No. US-2006-0061008; and/or Ser. No. 10/538,724, filedJun. 13, 2005 and published Mar. 9, 2006 as U.S. Publication No.US-2006-0050018, and/or U.S. provisional applications, Ser. No.61/466,138, filed Mar. 22, 2011; Ser. No. 61/452,816, filed Mar. 15,2011; and Ser. No. 61/426,328, filed Dec. 22, 2010, which are herebyincorporated herein by reference in their entireties.

Optionally, the display or displays and any associated user inputs maybe associated with various accessories or systems, such as, for example,a tire pressure monitoring system or a passenger air bag status or agarage door opening system or a telematics system or any other accessoryor system of the mirror assembly or of the vehicle or of an accessorymodule or console of the vehicle, such as an accessory module or consoleof the types described in U.S. Pat. Nos. 7,289,037; 6,877,888;6,824,281; 6,690,268; 6,672,744; 6,386,742; and 6,124,886, and/or U.S.patent application Ser. No. 10/538,724, filed Jun. 13, 2005 andpublished Mar. 9, 2006 as U.S. Publication No. US-2006-0050018, whichare hereby incorporated herein by reference in their entireties.

While the above description constitutes a plurality of embodiments ofthe present invention, it will be appreciated that the present inventionis susceptible to further modification and change without departing fromthe fair meaning of the accompanying claims.

1. A method of establishing a composite image for displaying in avehicle, comprising: a) providing a first camera and a second camera,each camera having a field of view; b) positioning the cameras so thatthe fields of view of the cameras overlap partially, wherein the camerastogether have a combined field of view; c) recording preliminary digitalimages from the cameras, each preliminary digital image being made up ofa plurality of pixels; and d) generating a final composite digital imagethat corresponds to a selected digital representation of the combinedfield of view of the cameras, at least in part by remapping selectedpixels from each of the preliminary digital images into selectedpositions of the final composite digital image.
 2. A method as claimedin claim 1, wherein the preliminary digital images each have associatedtherewith a preliminary apparent camera position and the final compositedigital image has associated therewith a final apparent camera position,and wherein the selected pixels from the preliminary digital images areselected so that the final apparent camera position associated with thefinal composite digital image is higher than the preliminary apparentcamera positions associated with the preliminary digital images.
 3. Amethod as claimed in claim 1, wherein the selected pixels from thepreliminary digital images are selected so that any misalignment betweenthe overlapping portions of the preliminary digital images issubstantially eliminated.
 4. A method as claimed in claim 1, wherein theselected pixels from the preliminary digital images are selected so thatthe final composite digital image is dewarped as compared to each of thepreliminary digital images.
 5. A method as claimed in claim 1, whereinthe preliminary digital images each have associated therewith apreliminary apparent camera position and the final composite digitalimage has associated therewith a final apparent camera position, andwherein the selected pixels from the preliminary digital images areselected so that the final apparent camera position associated with thefinal composite digital image is higher than the preliminary apparentcamera positions associated with the preliminary digital images, andwherein the selected pixels from the preliminary digital images areselected so that any misalignment between the overlapping portions ofthe preliminary digital images is substantially eliminated, and whereinthe selected pixels from the preliminary digital images are selected sothat the final composite digital image is dewarped as compared to eachof the preliminary digital images.
 6. A method as claimed in claim 1,further comprising: e) reading an input from a user of the vehicle priorto step d), and f) selecting a remapping table from a plurality ofremapping tables based on the input, wherein in step d) the finalcomposite digital image is generated using the remapping table selectedin step f).
 7. A method as claimed in claim 1, wherein step d) iscarried out in part by generating pixels in the final composite image byinterpolating between values of pixels that are adjacent to each other.8. A method as claimed in claim 1, further comprising: e) prior to stepd), selecting a subset of each preliminary digital image, and whereinselected pixels from each of the preliminary digital images are selectedfrom the subset of each preliminary digital image.
 9. A method ofestablishing a composite image for displaying in a vehicle, comprising:a) providing a first camera and a second camera, a third camera and afourth camera, each camera having a field of view, wherein the camerastogether have a combined field of view that is a 360 degree field ofview around the vehicle; b) positioning the cameras so that the field ofview of each camera overlaps partially with the field of view of two ofthe other cameras; c) recording preliminary digital images from thecameras, each preliminary digital image being made up of a plurality ofpixels; and d) generating a final composite digital image thatcorresponds to a selected digital representation of the combined fieldof view of the cameras by remapping selected pixels from each of thepreliminary digital images into selected positions of the finalcomposite digital image, wherein the preliminary digital images eachhave associated therewith a preliminary apparent camera viewpoint andthe final composite digital image has associated therewith a finalapparent camera viewpoint, and wherein the selected pixels from thepreliminary digital images are selected so that the final apparentcamera viewpoint associated with the final composite digital image ishigher than the preliminary apparent camera viewpoints associated withthe preliminary digital images, and wherein the selected pixels from thepreliminary digital images are selected so that any misalignment betweenthe overlapping portions of the preliminary digital images issubstantially eliminated, and wherein the selected pixels from thepreliminary digital images are selected so that the final compositedigital image is dewarped as compared to each of the preliminary digitalimages. 10-11. (canceled)
 12. A method of generating a remapping tablefor use in mapping pixels from a plurality of preliminary digital imagesinto a final composite image, comprising: a) driving a vehicle having afirst camera and a second camera thereon, each camera having a field ofview that overlaps partially with the field of view of the other camera,each camera having an imager for generating one of the preliminarydigital image, wherein the cameras together have a combined field ofview, wherein the vehicle further includes a controller; b) detecting atarget feature along the path of the vehicle during driving, using thecontroller; c) providing a first preliminary digital image from thefirst camera, wherein the first preliminary digital image contains afirst representation of the target feature at a first point time; d)determining the position of the first representation of the targetfeature in the first preliminary digital image; e) providing a secondpreliminary digital image from the second camera, wherein the secondpreliminary digital image contains a second representation of the targetfeature at a second point time; f) determining the position of thesecond representation of the target feature in the second preliminarydigital image; g) comparing the positions of the first and secondrepresentations of the target feature; and h) generating at least onevalue for the remapping table based on the result of the comparison instep g).
 13. A method as claimed in claim 12, further comprising: i)determining a direction of travel of the first representation of thetarget feature in the first preliminary digital image at the first pointin time; j) determining a direction of travel of the secondrepresentation of the target feature in the second preliminary digitalimage at the second point in time; k) comparing the directions of travelof the first and second representations of the target feature; and l)generating at least one value for the remapping table based on theresult of the comparison in step k).
 14. A method as claimed in claim13, further comprising: m) adjusting the values of selected pixels inthe remapping table by progressively smaller amounts based on thecomparison in step k).
 15. A method as claimed in claim 1, whereinpositioning the cameras comprises positioning the cameras at one of (i)a front of the vehicle so that the cameras have respective forwardfields of view ahead of the vehicle and (ii) a rear portion of thevehicle so that the cameras have respective forward fields of view aheadof the vehicle.
 16. A method as claimed in claim 1, comprising providingthird and fourth cameras, wherein the first, second, third and fourthcameras together have a combined field of view that is a 360 degreefield of view around the vehicle.
 17. A method as claimed in claim 16,wherein positioning the cameras comprises (i) positioning the firstcamera at a front of the vehicle so that the first camera has a forwardfield of view ahead of the vehicle, (ii) positioning the second cameraat a side of the vehicle so that the second camera has a sideward fieldof view, (iii) positioning the third camera at another side of thevehicle so that the third camera has a sideward field of view and (iv)positioning the fourth camera at a rear of the vehicle so that thefourth camera has a rearward field of view.
 18. A method as claimed inclaim 9, wherein positioning the cameras comprises (i) positioning thefirst camera at a front of the vehicle so that the first camera has aforward field of view ahead of the vehicle, (ii) positioning the secondcamera at a side of the vehicle so that the second camera has a sidewardfield of view, (iii) positioning the third camera at another side of thevehicle so that the third camera has a sideward field of view and (iv)positioning the fourth camera at a rear of the vehicle so that thefourth camera has a rearward field of view.
 19. A method as claimed inclaim 12, wherein the cameras are positioned at one of (i) a front ofthe vehicle so that the cameras have respective forward fields of viewahead of the vehicle and (ii) a rear portion of the vehicle so that thecameras have respective forward fields of view ahead of the vehicle. 20.A method as claimed in claim 12, wherein driving a vehicle comprisesdriving a vehicle having a first camera, a second camera, a third cameraand fourth camera thereon.
 21. A method as claimed in claim 20, whereinthe first, second, third and fourth cameras together have a combinedfield of view that is a 360 degree field of view around the vehicle. 22.A method as claimed in claim 21, wherein the first camera has a forwardfield of view forward of the vehicle, (ii) the second camera has asideward field of view sideward of a side of the vehicle, (iii) thethird camera has a rearward field of view rearward of the vehicle and(iv) the fourth camera has a sideward field of view sideward of theother side of the vehicle opposite the second camera.